Skip to content

fix(mcp): explain hasToken() separator errors in query tools - #3721

Merged
kodiakhq[bot] merged 5 commits into
mainfrom
brandon/mcp-text-index-needle
Nov 19, 2026
Merged

kodiakhq[bot] merged 5 commits into
mainfrom
brandon/mcp-text-index-needle

Conversation

@brandon-pereira

@brandon-pereira brandon-pereira commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Summary

MCP builder tools (clickstack_table, clickstack_search, clickstack_timeseries) failed about 50 times on prod in the week of Nov 9–16 with ClickHouse's "Needle must not contain whitespace or separator characters". Every one came from a SQL filter the agent wrote: hasToken(col, '<needle>') with a snake_case name, UUID, email, hostname or phrase. hasToken() only takes letters and digits; _ - . @ : and spaces are separators. The Lucene translation in queryParser.ts already splits terms on these characters and wasn't involved. None of the failing queries used hasAllTokens or the hasToken(lower(...)) shape it emits. clickstack_sql hit the same mistake about 17 times.

This PR doesn't change query generation. It helps the agent recover on the next call and avoid the mistake up front:

  • When this error comes back, the MCP error now ends with a hint: use a Lucene term (split into tokens, still uses the text index), or in SQL position(col, '…') > 0 for the exact term, which works on every ClickHouse version, with hasAllTokens(col, '…') AND as an optional prefix when the column has a text index. clickstack_sql gets only the SQL form, since it has no whereLanguage.
  • The hint also fires on any BAD_ARGUMENTS (code 36) error that mentions hasToken, so a reworded ClickHouse message doesn't silently drop it.
  • The shared whereLanguage description (used unchanged by all three builder tools) warns that hasToken rejects these needles.

An integration test runs a separator needle through clickstack_search against real ClickHouse and checks for the hint, so it fails if a future ClickHouse image changes the error enough to lose it. Rows are seeded because ClickHouse only checks the needle while it reads rows.

The first call still fails; only the retry is helped. Rewriting agent-written hasToken() calls before running them would stop the first failure, but rewriting user SQL with pattern matching is fragile, so it's left out.

How to reproduce

Starting point: yarn dev, any log source, an MCP client connected to the local API.

On main:

  1. Call clickstack_search with whereLanguage: "sql" and where: "hasToken(Body, 'payment_failed')" over a time range that has log rows.
  2. The error is ClickHouse's raw "Needle must not contain whitespace or separator characters" with no guidance.

On this branch:

  1. Same call.
  2. The error ends with HINT: hasToken() takes a single token of letters and digits; ..., pointing to Lucene or position().

Tests

  • Unit (query.test.ts): current message gets the hint; reworded message with BAD_ARGUMENTS still gets it; an unrelated BAD_ARGUMENTS error does not.
  • Integration (queryTool.int.test.ts): 130/130 pass locally on ClickHouse 26.10. Prod customers were on 26.6, which uses the same message.

@brandon-pereira
brandon-pereira requested a review from a team as a code owner November 18, 2026 19:34
@vercel

vercel Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 19, 2026 3:55pm UTC
hyperdx-storybook Ready Ready Preview Nov 19, 2026 3:55pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: a44ee81

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 P0/P1 in code this PR adds or changes: fix before asking for review.
  • 🟡 P2 in code this PR changes: your call. Fix if small, otherwise reply.
  • 🔵 P3 is collapsed at the bottom: nobody needs to act on it before merge.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    priority. Reply with one sentence; if it is P0, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

2 finding(s): 🔴 0 P0/P1 · 🟡 2 P2 · 🔵 0 P3

1 posted as inline comment(s) on the changed lines. 1 unchanged from an earlier push (already inline above).


Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

🔵 Tier 2 — Low Risk

Small, isolated change with no API route or data model modifications.

Why this tier:

  • Standard feature/fix — introduces new logic or modifies core functionality

Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns.
SLA: Resolve within 4 business hours.

Stats
  • Production files changed: 2
  • Production lines changed: 36 (+ 79 in test files, excluded from tier calculation)
  • Branch: brandon/mcp-text-index-needle
  • Author: brandon-pereira

To override this classification, remove the review/tier-2 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 107 in total (1.12%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. This is a small, well-tested, behavior-additive change: the new errorHint() branch only decorates error text and never affects query execution. Branch ordering, the Code: 36\b boundary, position(haystack, needle) argument order, and the getClickHouseErrorType cause-chain lookup all check out against the code. No P0/P1/P2 findings.

🔵 P3 nitpicks (2)
  • packages/api/src/mcp/__tests__/query.test.ts:352 — the Code: 36\b alternative in the new hint branch is never exercised in isolation; the only test carrying Code: 36 also contains the literal BAD_ARGUMENTS, so it passes via the other alternation and a regression in the Code: 36\b clause (or its \b boundary) would survive.
    • Fix: Add a test whose message contains hasToken and Code: 36 but not BAD_ARGUMENTS asserting the hint fires, plus a Code: 360-style case asserting it does not.
    • testing
  • packages/api/src/mcp/tools/query/helpers.ts:1054 — the separator-character set and "use Lucene" guidance are restated in three spots (the branch comment, the hint text, and WHERE_LANGUAGE_DESCRIPTION in schemas.ts) with no shared source, so they can drift if ClickHouse's separator list or the recommended recovery changes.
    • Fix: Extract the separator list (and ideally the one-line guidance) into a shared exported constant imported by both files.
    • maintainability

Reviewers (5): correctness, testing, maintainability, kieran-typescript, project-standards.

Testing gaps:

  • The getClickHouseErrorType path for a direct ClickHouseError passed as error (type on the error itself, not via .cause) is not covered; only the .cause-wrapped path is exercised.
  • The raw-SQL position()/hasAllTokens() hint (the clickstack_sql path) is covered only by unit tests, not by the real-ClickHouse integration test that guards the Lucene path.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 481 passed • 1 skipped • 1104s

Status Count
✅ Passed 481
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

) {
// position() works on every ClickHouse version and, unlike LIKE, has no '_'
// wildcard. hasAllTokens() is newer and only worth adding with a text index.
const sqlAlternatives =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — The SQL fix in the hint always uses the literal 'payment_failed', whatever needle actually failed

The ClickHouse message already names the failing needle (FUNCTION hasToken(<col> : 0, '<needle>'_String ...)). Pull it out with a regex like /hasToken\w*\([^,]+,\s*'((?:[^'\\]|\\.)*)'/ and put it into the position()/hasAllTokens() snippet, falling back to '<term>' when nothing matches. Right now an agent whose needle was a UUID or an email is told to "Use position(col, 'payment_failed') > 0 for the exact term", and in the rawSql branch there isn't even an "e.g." to show it's only an example.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

// '@', ':' and spaces are separators. Agents reach for it in SQL filters
// with ids, emails and snake_case names. The BAD_ARGUMENTS + hasToken
// fallback keeps the hint if ClickHouse rewords the message.
if (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — A hasTokenCaseInsensitive() failure gets advice that changes the match to case-sensitive

hasTokenCaseInsensitive throws the same "Needle must not contain..." message, so the first regex catches it. The hint then calls the function hasToken() and recommends position(col, ...), which matches case-sensitively, so following it can silently miss rows the original query was meant to find. When the message contains hasTokenCaseInsensitive, recommend positionCaseInsensitive(col, ...) instead. The \bhasToken\b fallback also misses this function, so the hint would disappear for it if ClickHouse rewords the message.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

// fallback keeps the hint if ClickHouse rewords the message.
if (
/Needle must not contain whitespace or separator characters/i.test(msg) ||
(/\bhasToken\b/.test(msg) &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — Fallback fires on any BAD_ARGUMENTS error whose text mentions hasToken, not only on hasToken's own failure

The fallback only checks that hasToken appears somewhere in the message. ClickHouse analyzer errors often echo the whole expression (In scope SELECT ... WHERE ...), and Lucene filters compile to hasToken(lower(col), '...'). So a BAD_ARGUMENTS error from another function in a Lucene query (e.g. splitByChar in a select expression) gets this hint, telling the agent to switch to Lucene when it already uses Lucene. Fix: anchor the fallback to the failing function, e.g. /FUNCTION hasToken(CaseInsensitive)?\(/ or /function hasToken\b/i. Add a test with a non-hasToken BAD_ARGUMENTS message that still contains hasToken( in its scope text.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@kodiakhq
kodiakhq Bot merged commit e80fc1b into main Nov 19, 2026
28 checks passed
@kodiakhq
kodiakhq Bot deleted the brandon/mcp-text-index-needle branch November 19, 2026 15:59

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — a44ee818 Deployed Nov 19, 2026 by vercel[bot]
Preview – hyperdx-storybook — a44ee818 Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge review/tier-2 Low risk — AI review + quick human skim

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants